Three-dimensional (3D) television has been a technology trend in recent years that intends to bring viewers sensational viewing experience. Various technologies have been developed to enable 3D viewing. Among them, the multi-view video is a key technology for 3DTV application among others. The traditional video is a two-dimensional (2D) medium that only provides viewers a single view of a scene from the perspective of the camera. However, the multi-view video is capable of offering arbitrary viewpoints of dynamic scenes and provides viewers the sensation of realism.
The multi-view video is typically created by capturing a scene using multiple cameras simultaneously, where the multiple cameras are properly located so that each camera captures the scene from one viewpoint. Accordingly, the multiple cameras will capture multiple video sequences corresponding to multiple views. In order to provide more views, more cameras have been used to generate multi-view video with a large number of video sequences associated with the views. Accordingly, the multi-view video will require a large storage space to store and/or a high bandwidth to transmit. Therefore, multi-view video coding techniques have been developed in the field to reduce the required storage space or the transmission bandwidth.
A straightforward approach may be to simply apply conventional video coding techniques to each single-view video sequence independently and disregard any correlation among different views. Such coding system would be very inefficient. In order to improve efficiency of multi-view video coding, typical multi-view video coding exploits inter-view redundancy. Therefore, most 3D Video Coding (3DVC) systems take into account of the correlation of video data associated with multiple views and depth maps. The standard development body, the Joint Video Team of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG), extended H.264/MPEG-4 AVC to multi-view video coding (MVC) for stereo and multi-view videos.
The MVC adopts both temporal and spatial predictions to improve compression efficiency. During the development of MVC, some macroblock-level coding tools are proposed, including illumination compensation, adaptive reference filtering, motion skip mode, and view synthesis prediction. These coding tools are proposed to exploit the redundancy between multiple views. Illumination compensation is intended for compensating the illumination variations between different views. Adaptive reference filtering is intended to reduce the variations due to focus mismatch among the cameras. Motion skip mode allows the motion vectors in the current view to be inferred from the other views. View synthesis prediction is applied to predict a picture of the current view from other views.
In the reference software for HEVC based 3D video coding (3D-HTM), inter-view candidate is added as a motion vector (MV) or disparity vector (DV) candidate for Inter, Merge and Skip mode in order to re-use previously coded motion information of adjacent views. In 3D-HTM, the basic unit for compression, termed as coding unit (CU), is a 2N×2N square block. Each CU can be recursively split into four smaller CUs until a predefined minimum size is reached. Each CU contains one or more prediction units (PUs).
To share the previously coded texture information of adjacent views, a technique known as Disparity-Compensated Prediction (DCP) has been included in 3D-HTM as an alternative coding tool to motion-compensated prediction (MCP). MCP refers to an inter-picture prediction that uses previously coded pictures of the same view, while DCP refers to an inter-picture prediction that uses previously coded pictures of other views in the same access unit. FIG. 1 illustrates an example of 3D video coding system incorporating MCP and DCP. The vector (110) used for DCP is termed as disparity vector (DV), which is analog to the motion vector (MV) used in MCP. FIG. 1 illustrates three MVs (120, 130 and 140) associated with MCP. Moreover, the DV of a DCP block can also be predicted by the disparity vector predictor (DVP) candidate derived from neighboring blocks or the temporal collocated blocks that also use inter-view reference pictures. In 3D-HTM, when deriving an inter-view Merge candidate for Merge/Skip modes, if the motion information of corresponding block is not available or not valid, the inter-view Merge candidate is replaced by a DV.
Inter-view residual prediction is another coding tool used in 3D-HTM. To share the previously coded residual information of adjacent views, the residual signal of the current prediction block (PU) can be predicted by the residual signals of the corresponding blocks in the inter-view pictures as shown in FIG. 2. The corresponding blocks can be located by respective DVs. The video pictures and depth maps corresponding to a particular camera position are indicated by a view identifier (i.e., V0, V1 and V2 in FIG. 2). All video pictures and depth maps that belong to the same camera position are associated with the same viewId (i.e., view identifier). The view identifiers are used for specifying the coding order within the access units and detecting missing views in error-prone environments. An access unit includes all video pictures and depth maps corresponding to the same time instant. Inside an access unit, the video picture and, when present, the associated depth map having viewId equal to 0 are coded first, followed by the video picture and depth map having viewId equal to 1, etc. The view with viewId equal to 0 (i.e., V0 in FIG. 2) is also referred to as the base view or the independent view. The base view video pictures can be coded using a conventional HEVC video coder without dependence on other views.
As can be seen in FIG. 2, for the current block, motion vector predictor (MVP)/disparity vector predictor (DVP) can be derived from the inter-view blocks in the inter-view pictures. In the following, inter-view blocks in inter-view picture may be abbreviated as inter-view blocks. The derived candidate is termed as inter-view candidates, which can be inter-view MVPs or DVPs. The coding tools that codes the motion information of a current block (e.g., a current prediction unit, PU) based on previously coded motion information in other views is termed as inter-view motion parameter prediction. Furthermore, a corresponding block in a neighboring view is termed as an inter-view block and the inter-view block is located using the disparity vector derived from the depth information of current block in current picture.
View synthesis prediction (VSP) is a technique to remove interview redundancies among video signal from different viewpoints, in which synthetic signal is used as references to predict a current picture. In 3D-HEVC test model, there exists a process to derive a disparity vector predictor. The derived disparity vector is then used to fetch a depth block in the depth image of the reference view. The fetched depth block would have the same size of the current prediction unit (PU), and it will then be used to do backward warping for the current PU. In addition, the warping operation may be performed at a sub-PU level precision, like 8×4 or 4×8 blocks. A maximum depth value is picked for a sub-PU block and used for warping all the pixels in the sub-PU block. The VSP technique is applied for texture picture coding. In current implementation, VSP is added as a new merging candidate to signal the use of VSP prediction. In such a way, a VSP block may be a skipped block without any residual, or a merge block with residual information coded.
The example shown in FIG. 2 corresponds to a view coding order from V0 (i.e., base view), V1, and V2. The current block in the current picture being coded is in V2. According to HTM3.1, all the MVs of reference blocks in the previously coded views can be considered as an inter-view candidate even if the inter-view pictures are not in the reference picture list of current picture. In FIG. 2, frames 210, 220 and 230 correspond to a video picture or a depth map from views V0, V1 and V2 at time t1 respectively. Block 232 is the current block in the current view, and blocks 212 and 222 are the current blocks in V0 and V1 respectively. For current block 212 in V0, a disparity vector (216) is used to locate the inter-view collocated block (214). Similarly, for current block 222 in V1, a disparity vector (226) is used to locate the inter-view collocated block (224). According to HTM3.1, the motion vectors or disparity vectors associated with inter-view collocated blocks from any coded views can be included in the inter-view candidates. Therefore, the number of inter-view candidates can be rather large, which will require more processing time and large storage space. It is desirable to develop a method to reduce the processing time and or the storage requirement without causing noticeable impact on the system performance in terms of BD-rate or other performance measurement.
In 3DV-HTM, a disparity vector can be used as a DVP candidate for Inter mode or as a Merge candidate for Merge/Skip mode. A derived disparity vector can also be used as an offset vector for inter-view motion prediction and inter-view residual prediction. When used as an offset vector, the DV is derived from spatial or temporal neighboring blocks as shown in FIG. 3. Multiple spatial and temporal neighboring blocks are determined and DV availability of the spatial and temporal neighboring blocks is checked according to a pre-determined order. This coding tool for DV derivation based on neighboring (spatial and temporal) blocks is termed as Neighboring Block DV (NBDV). As shown in FIG. 3A, the spatial neighboring block set includes the location diagonally across from the lower-left corner of the current block (i.e., A0), the location next to the left-bottom side of the current block (i.e., A1), the location diagonally across from the upper-left corner of the current block (i.e., B2), the location diagonally across from the upper-right corner of the current block (i.e., B0), and the location next to the top-right side of the current block (i.e., B1). As shown in FIG. 3B, the temporal neighboring block set includes the location at the center of the current block (i.e., BCTR) and the location diagonally across from the lower-right corner of the current block (i.e., RB) in a temporal reference picture. Instead of the center location, other locations (e.g., a lower-right block) within the current block in the temporal reference picture may also be used. In other words, any block collocated with the current block can be included in the temporal block set. Once a block is identified as having a DV, the checking process will be terminated. An exemplary search order for the spatial neighboring blocks in FIG. 3A is (A1, B1, B0, A0, B2). An exemplary search order for the temporal neighboring blocks for the temporal neighboring blocks in FIG. 3B is (BR, BCTR). The spatial and temporal neighboring blocks are the same as the spatial and temporal neighboring blocks of AMVP and Merge modes in HEVC.
If a DCP coded block is not found in the neighboring block set (i.e., spatial and temporal neighboring blocks as shown in FIGS. 3A and 3B), the disparity information can be obtained from another coding tool (DV-MCP). In this case, when a neighboring block is MCP coded block and its motion is predicted by the inter-view motion prediction, as shown in FIG. 4, the disparity vector used for the inter-view motion prediction represents a motion correspondence between the current and the inter-view reference picture. This type of motion vector is referred to as inter-view predicted motion vector and the blocks are referred to as DV-MCP blocks. FIG. 4 illustrates an example of a DV-MCP block, where the motion information of the DV-MCP block (410) is predicted from a corresponding block (420) in the inter-view reference picture. The location of the corresponding block (420) is specified by a disparity vector (430). The disparity vector used in the DV-MCP block represents a motion correspondence between the current and inter-view reference picture. The motion information (422) of the corresponding block (420) is used to predict motion information (412) of the current block (410) in the current view.
To indicate whether a MCP block is DV-MCP coded and to store the disparity vector for the inter-view motion parameters prediction, two variables are used to represent the motion vector information for each block:                dvMcpFlag, and        dvMcpDisparity.        
When dvMcpFlag is equal to 1, the dvMcpDisparity is set to indicate that the disparity vector is used for the inter-view motion parameter prediction. In the construction process for the AMVP and Merge candidate list, the dvMcpFlag of the candidate is set to 1 if the candidate is generated by inter-view motion parameter prediction and is set to 0 otherwise. The disparity vectors from DV-MCP blocks are used in following order: A0, A1, B0, B1, B2, Co1 (i.e., Collocated block, BCTR or RB).
A method to enhance the NBDV by extracting a more accurate disparity vector from the depth map is utilized in current 3D-HEVC. A depth block from coded depth map in the same access unit is first retrieved and used as a virtual depth of the current block. This coding tool for DV derivation is termed as Depth-oriented NBDV (DoNBDV). While coding the texture in view 1 and view 2 with the common test condition, the depth map in view 0 is already available. So the coding of texture in view 1 and view 2 can be benefited from the depth map in view 0. An estimated disparity vector can be extracted from the virtual depth shown in FIG. 5. The overall flow is as following:
1. Use an estimated disparity vector, which is the NBDV in current 3D-HTM, to locate the corresponding block in the coded texture view
2. Use the collocated depth in the coded view for current block (coding unit) as virtual depth.
3. Extract a disparity vector for inter-view motion prediction from the maximum value in the virtual depth retrieved in the previous step.
In the example illustrated in FIG. 5, the coded depth map in view 0 is used to derive the DV for the texture frame in view 1 to be coded. A corresponding depth block (530) in the coded DO is retrieved for the current block (CB, 510) according to the estimated disparity vector (540) and the location (520) of the current block of the coded depth map in view 0. The retrieved block (530) is then used as the virtual depth block (530′) for the current block to derive the DV. The maximum value in the virtual depth block (530′) is used to extract a disparity vector for inter-view motion prediction.
In current 3D-AVC (3D video coding based on Advanced Video Coding (AVC)), the disparity vector (DV) is used for disparity compensated prediction (DCP), predicting DV and indicating the inter-view corresponding block to derive inter-view candidate.
In order to share the previously encoded texture information of reference views, the concept of disparity-compensated prediction (DCP) has been added as an alternative to the motion-compensated prediction (MCP). MCP refers to an inter picture prediction that uses already coded pictures of the same view in a different access unit, while DCP refers to an inter picture prediction that uses already coded pictures of other views in the same access unit. The vector used for DCP is termed disparity vector (DV), which is analog to the motion vector (MV) used in MCP.
In Inter mode, Direction-Separate Motion Vector Prediction is another coding tool used in 3D-AVC. The direction-separate motion vector prediction consists of the temporal and inter-view motion vector prediction. If the target reference picture is a temporal prediction picture, the temporal motion vectors of the adjacent blocks around the current block Cb, such as A, B, and C in FIG. 6A are employed in the derivation of the motion vector prediction. If a temporal motion vector is unavailable, an inter-view motion vector is used. The inter-view motion vector is derived from the corresponding block indicated by a DV converted from depth. The motion vector prediction is then derived as the median of the motion vectors of the adjacent blocks A, B, and C. Block D is used only when C is unavailable.
On the contrary, if the target reference picture is an inter-view prediction picture, the inter-view motion vectors of the neighboring blocks are employed for the inter-view prediction. If an inter-view motion vector is unavailable, a disparity vector which is derived from the maximum depth value of four corner depth samples within the associated depth block is used. The motion vector predictor is then derived as the median of the inter-view motion vector of the adjacent blocks A, B, and C.
When the target reference picture is an inter-view prediction picture, the inter-view motion vectors of the neighboring blocks are used to derive the inter-view motion vector predictor. In block 610 of FIG. 6B, inter-view motion vectors of the spatially neighboring blocks are derived based on the texture data of respective blocks. The depth map associated with the current block Cb is also provided in block 660. The availability of inter-view motion vector for blocks A, B and C is checked in block 620. If an inter-view motion vector is unavailable, the disparity vector for the current block is used to replace the unavailable inter-view motion vector as shown in block 630. The disparity vector is derived from the maximum depth value of the associated depth block as shown in block 670. The median of the inter-view motion vectors of blocks A, B and C is used as the inter-view motion vector predictor. The conventional MVP procedure, where a final MVP is derived based on the median of the motion vectors of the inter-view MVPs or temporal MVPs as shown in block 640. Motion vector coding based on the motion vector predictor is performed as shown in block 650.
Priority based MVP candidate derivation for Skip/Direct mode is another coding tool for 3D-AVC. In Skip/Direct mode, a MVP candidate is derived based on a predefined derivation order: inter-view candidate and the median of three spatial candidates derived from the neighboring blocks A, B, and C (D is used only when C is unavailable) as shown in FIG. 7.
Inter-view MV candidate derivation is also shown in FIG. 7. The central point (712) of the current block (710) in the dependent view and its disparity vector are used to find a corresponding point in the base view or reference view. After that, the MV of the block including the corresponding point in the base view is used as the inter-view candidate of the current block. The disparity vector can be derived from both the neighboring blocks (A, B and C/D) and the depth value of the central point. Specifically, if only one of the neighboring blocks has disparity vector (DV), the DV is used as the disparity. Otherwise, the DV is then derived as the median of the DVs (720) of the adjacent blocks A, B, and C. If a DV is unavailable, a DV converted from depth is then used instead. The derived DV is used to locate a corresponding block (740) in the reference picture (730).
As described above, DV derivation is critical in 3D video coding for both 3D-HEVC and 3D-AVC. Considering the condition that the testing 3D sequences are rectified and no vertical shifting between views, some coding tools may only use the DV without vertical component for inter-view data access. However, when the input data is not rectified, we may still need the vertical component of the DV to indicate the correct corresponding block in the other views.